feat(triage): add effort-based gating via effort-estimation skill#36
feat(triage): add effort-based gating via effort-estimation skill#36rh-hemartin wants to merge 1 commit into
Conversation
|
🤖 Review · |
PR Summary by QodoAdd effort scoring and auto-promotion gating to triage pipeline
AI Description
Diagram
High-Level Assessment
Files changed (4)
|
Code Review by Qodo
Context used✅ Compliance rules (platform):
55 rules 1. Protected scripts/ files modified
|
|
🤖 Finished Review · ✅ Success · Started 6:35 AM UTC · Completed 6:44 AM UTC |
ReviewFindingsMedium
Low
Previous runReviewFindingsMedium
Low
Labels: PR modifies triage agent prompt, post-script, and adds effort-estimation skill to the triage pipeline. Previous run (2)ReviewFindingsMedium
Low
Previous run (3)ReviewFindingsMedium
Low
Previous run (4)ReviewVerdict: comment · 2 medium, 2 low findings The new head (09a16a7) is a force-push from the previously-reviewed 8408ffc. The key code change: Three prior low findings are resolved:
The effort-gating logic is correctly implemented: FindingsMedium
Low
Previous run (5)ReviewVerdict: approve · 0 medium, 5 low findings The new head (8408ffc) is a force-push from b2b78ae that addresses all three prior medium findings:
The effort-gating logic is correctly implemented: Findings1. No test for effort just below threshold · low · test-adequacyFile: The test suite covers effort=0.25, 1.0 (below threshold) and effort=2.0, 2.5, 3.0 (at/above threshold), but no test verifies effort=1.5 or 1.99 routes to Remediation: Add a test case with effort=1.5 or 1.99 asserting 2. Test fixtures for non-applicable categories include effort values · low · test-integrityFile: The Remediation: Remove effort from feature/other fixtures, or add a comment explaining they verify effort is ignored for non-applicable categories. 3. Schema validation tests don't cover conditional effort requirement · low · test-adequacyFile: The schema validation test suite has one Remediation: Add schema validation test cases: a bug category without effort that fails, and a feature category without effort that passes. 4. No linked issue for non-trivial feature change · low · authorizationFile: The PR references #2207 in a code comment but no issue URL is linked in the PR metadata. The PR body states it was "Migrated from fullsend-ai/fullsend feat/triage-effort-analysis", providing implicit authorization. 5. Triage effort estimation overlaps with prioritize agent's RICE effort · low · architectural-coherenceFile: The triage agent now estimates effort (0.25–3 scale) matching the prioritize agent's RICE effort dimension. Triage effort gates auto-promotion (< 2.0 →
Previous run (6)ReviewVerdict: comment · 3 medium, 5 low findings The new head (b2b78ae) is a force-push from the previously-reviewed 8e5ff3f. The effort-gating code across all four files is functionally identical to the prior review. All eight prior findings remain unaddressed. The effort-gating logic is correctly implemented: Findings1. GHA workflow command injection via unsanitized EFFORT interpolation · medium · securityFile: Two interpolation points expose unsanitized
Note: This is defense-in-depth — schema validation ( Remediation: Sanitize EFFORT before all SAFE_EFFORT="${EFFORT//::/}"
SAFE_EFFORT="${SAFE_EFFORT//%0A/}"
SAFE_EFFORT="${SAFE_EFFORT//%0a/}"
SAFE_EFFORT="${SAFE_EFFORT//%0D/}"
SAFE_EFFORT="${SAFE_EFFORT//%0d/}"2. Documentation now contradicts actual behavior —
|
waynesun09
left a comment
There was a problem hiding this comment.
Multi-agent review squad (5 agents: Claude, Gemini, Codex) findings — Medium and above, deduplicated and verified against the head branch. Two additional Medium findings below couldn't be attached inline since they target files/state outside this PR's diff:
[MEDIUM] docs/triage.md control-label documentation is now stale — docs/triage.md:40-41 (not touched by this PR) still describes ready-to-code as "low-risk (bug, documentation, performance)" and triaged as "feature or other category," with no mention of the new effort threshold. docs/code.md:41 has the same staleness. Already flagged independently by the qodo-code-review bot on this PR ("Docs contradict effort gating").
[MEDIUM] Branch is 24 commits behind main; CI test suite has never run against it — git compare main...feat/triage-effort-analysis shows the branch diverged and is missing main's 88800b2 (adds a label-contradiction guard to the same sufficient) case block this PR modifies) plus the Makefile/script-test.yml CI infra added after this branch was cut. No script-test check has run on this PR — only dispatch/dispatch and DCO. Suggest rebasing onto current main before merge and confirming the effort-gating logic composes cleanly with the label-contradiction guard.
Four inline comments above cover the remaining High/Medium findings (stale triaged label bug, effort-scale mismatch with prioritize.md, optional-but-load-bearing effort schema field, and the round-down bias prompt-injection surface).
c51787d to
24b4069
Compare
|
🤖 Finished Review · ✅ Success · Started 6:26 PM UTC · Completed 6:35 PM UTC |
24b4069 to
2856f9d
Compare
|
🤖 Finished Review · ✅ Success · Started 6:38 PM UTC · Completed 6:51 PM UTC |
2856f9d to
60d8500
Compare
|
🤖 Finished Review · ✅ Success · Started 7:13 PM UTC · Completed 7:28 PM UTC |
|
🤖 Review · |
|
🤖 Review · |
|
🤖 Finished Review · ✅ Success · Started 8:02 PM UTC · Completed 8:11 PM UTC |
213c3b9 to
39c1f2a
Compare
|
🤖 Review · |
8408ffc to
137354b
Compare
|
🤖 Review · |
137354b to
2e5d15d
Compare
|
🤖 Review · |
2e5d15d to
09a16a7
Compare
|
🤖 Finished Review · ✅ Success · Started 8:15 AM UTC · Completed 8:29 AM UTC |
09a16a7 to
39a75aa
Compare
|
🤖 Finished Review · ✅ Success · Started 6:47 AM UTC · Completed 7:05 AM UTC |
waynesun09
left a comment
There was a problem hiding this comment.
Multi-agent review squad pass (models: Claude, Grok), deduplicated against all prior review threads and verified against head 39a75aa. All eight July 15 fixes are confirmed correctly in place, and both test suites pass at head. Posting 3 new MEDIUM findings inline; a 4th MEDIUM is an escalation of the existing unreplied July 20 performance-label thread and was posted as a reply there to keep the discussion in one place.
39a75aa to
350f215
Compare
|
🤖 Review · |
350f215 to
3ca1484
Compare
|
🤖 Finished Review · ✅ Success · Started 10:31 AM UTC · Completed 10:48 AM UTC |
waynesun09
left a comment
There was a problem hiding this comment.
Multi-agent review squad pass (models: Claude, Grok) at head 3ca1484, deduplicated against all prior review threads. All 11 previously fixed findings are confirmed in place, and both test suites pass at head (76 tests). Posting 3 new MEDIUM findings inline. The three still-unreplied threads (silent performance label auto-creation, docs/triage.md staleness, dual effort-estimation systems) were independently rediscovered by this round's agents but are not re-posted here — see their existing threads.
3ca1484 to
0829644
Compare
|
🤖 Finished Review · ✅ Success · Started 6:55 AM UTC · Completed 7:10 AM UTC |
Introduce a new skill that users can override to control how effort is scored and when issues require human review before auto-promoting to the coder agent. The skill scores implementation effort on a 0.25-3 scale and returns an effort_requires_review boolean that the post-script uses to route issues to human review or auto-dispatch. The skill owns the scoring table, signals, threshold, and estimation rules. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Hector Martinez <hemartin@redhat.com>
0829644 to
b842d8c
Compare
|
🤖 Finished Review · ✅ Success · Started 8:51 AM UTC · Completed 9:08 AM UTC |
Superseded by updated review
| echo "Resetting triage labels on ${REPO}#${ISSUE_NUMBER}" | ||
|
|
||
| for label in needs-info ready-to-code duplicate feature question not-planned; do | ||
| for label in needs-info ready-to-code duplicate feature question not-planned triaged; do |
There was a problem hiding this comment.
[low] label-reset-consistency
The label reset for-loop and jq verification filter enumerate label names separately. Adding triaged to both is correct but extends a pre-existing maintenance burden where the two lists can diverge silently.
Summary
effort-estimationskill that scores implementation effort (0.25-3 scale) and returns aneffort_requires_reviewboolean controlling whether issues auto-promote to the coder or wait for human revieweffort_requires_reviewinstead of comparing against a hardcoded thresholdCloses fullsend-ai/fullsend#2207
Test plan
bash scripts/post-triage-test.sh-- all 68 cases passeffort-estimationin skills arrayeffort_requires_reviewboolean, not numeric thresholdeffort,effort_requires_review)